Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency

Sample efficiency is crucial for imitation learning methods to be applicable in real-world applications. Many studies improve sample by extending adversarial off-policy regardless of the fact that these extensions could either change original objective or involve complicated optimization. We revisit foundation and propose an efficient approach requires no training min-max Our formulation capitalizes on two key insights: (1) similarity between Bellman equation stationary state-action distribution allows us derive a novel temporal difference (TD) approach; (2) use deterministic policy simplifies TD learning. Combined, insights yield practical algorithm, Deterministic Discriminative Imitation (D2-Imitation), which oper- ates first partitioning samples into replay buffers then via reinforcement empirical results show D2-Imitation effective achieving good efficiency, outperforming several extension approaches many control tasks.